Classifying Code Changes and Predicting Defects Using Change Genealogies

Identifying bug fixes and using them to estimate or even predict software quality is a frequent task when mining version archives. The number of applied bug fixes serves as code quality metric identifying defect-prone and non-defect-prone code artifacts. But when is a set of applied code changes, we call it change set, considered a bug fix and which metrics should be used to building high quality defect prediction models? Most commonly, bug fixes are identified by analyzing commit messages—short, mostly unstructured pieces of plain text. Commit message containing keywords such as “fix” or “issue” followed by a bug report identifier, are considered to fix the corresponding bug report. Similar, most defect prediction models use metrics describing the structure, complexity, or dependencies of source code artifacts. Complex or central code is considered to be more defect-prone. But commit messages and code metrics describe the state of software artifacts and code changes at a particular point in time, disregarding their genealogies that describe how the current state description came to be. There are approaches measuring historic properties of code artifacts [1]–[5] and using code dependency graphs [6], [7] but non of these approaches tracks the structural dependency paths of code changes to measure the centrality and impact of change sets, although change sets are those development events that make the source code look as it does. Herzig et al. [8] used so called change genealogy graphs to model structural dependencies between change sets. The authors used these change genealogy graphs to measure and analyze the impact of change sets on other, later applied change sets. In this paper, we make use of change genealogy graphs to define a set of change genealogy network metrics describing the structural dependencies of change sets. We further investigate whether change genealogy metrics can be used to identify bug fixing change sets (without using commit messages and bug databases) and whether change genealogy metrics are expressive enough to build effective defect prediction models classifying source files to be defect-prone or not. Regarding the identification of bug fixing change sets, our assumption is that change sets applying bug fixes show significant dependency differences when compared to change sets applying new feature implementations. We suspect that implementing and adding a new feature implies adding new method definitions that impact a large set of later applied code changes, which add code fragments adding method calls to these newly defined methods. In contrast, we suspect bug fixes to be relatively small rarely defining new methods but modifying existing features and thus to have a small impact on later applied code changes. The impact of bug fixes is to modify the runtime behavior of the software system rather than causing future change sets to use different functionality. Similar, we suspect more central change sets—depending on a large set of earlier change sets and causing many later applied change sets to be dependent on itself—to be crucial to the software development process. Consequently, we suspect code artifacts that got many crucial and central code changes applied to be more defect prone than others. More specifically, we seek to answer the following research questions in our study: RQ1 How do bug fix classification models based on change genealogy metrics compare to classification models based on code complexity metrics (Section V)? RQ2 How do defect prediction models compare with defect prediction models based on code complexity or code dependency network metrics (Section VI)? We tested the classification and prediction abilities of our approaches on four open source projects. The results show that change genealogy metrics can be used to separate bug fixing from feature implementing change sets with an average precision of 72% and an average recall of 89%. Our results also show that defect prediction models based on change genealogy metrics can predict defect-prone source files with precision and recall values of up to 80%. On average the precision for change genealogy models lies at 69% and the average recall at 81%. Compared to prediction models based on code dependency network metrics, change genealogy based prediction models achieve better precision and comparable recall values.